Auto depth field capturing system and method thereof

ABSTRACT

The invention presents a system and method for obtaining object depth through digital signal processing. The auto depth-field capturing method for a camera includes the steps of a) taking plural images; b) estimating plural epipolar data of the plural images for obtaining a matrix describing motion and directional vectors; c) estimating a location data in response to the plural epipolar data and the matrix; d) rectifying the plural images corresponding to the plural epipolar data for obtaining plural rectified images; e) calculating the location data for obtaining disparity vectors of the rectified images; f) obtaining a depth map in response to the disparity vectors and the location data; and g) painting a 3D image in correspondence with the depth map. The depth estimation method of the present invention is fully automatic without change of the camera itself.

FIELD OF THE INVENTION

The present invention relates to an image capturing system, and moreparticularly, to an auto depth field capturing system for a camera and amethod thereof.

BACKGROUND OF THE INVENTION

The television has been developed from the beginning of 20th century,wherein the black and white one, the color one, and even the digitaltelevision were disclosed for continuously progresses. Human beings keepchallenging to improve the science and technology for developing thebetter vision. In 21st century, people still strive for developing newdisplaying technique, wherein the new generation displayer could providemore colorful and finer vision.

According to the prior art, the display, such as CRT TV, PC monitor, LCDTV, and PDP TV, is based on 2-D displaying technique. However, the humanvision is based on stereoscopy. For achieving the purpose ofstereoscopy, it is important to estimate the depth of objects while thestereo image is taken by a camera. For solving the above problem, U.S.Pat. No. 6,959,253 described a method for calibrating machine visionmeasuring systems that have more than one camera. Furthermore, U.S. Pat.No. 6,781,618 discloses a method consisting of the construction of a 3Dscene model by acquiring first images of a scene having unknowncharacteristics with a first camera. Corresponding second images ofanother scene having known characteristics are acquired by a secondcamera. The first and second cameras have a fixed physical relationshipto each other. The 3D model should be analyzed by means of using thecorresponding positions and the fixed physical relationship of twocameras.

Please refer to FIG. 1. It illustrates a block diagram showing theelectrical construction of the stereo-image capturing device of U.S.Pat. No. 6,977,674. As shown in FIG. 1, a CCD with the RGB on-chip colorfilter 14 attached is utilized for the imaging device 11. Namely, thecolor filters which are applied to the apertures 22R, 22G, and 22Bcorrespond to the color filters which are utilized for the imagingdevice 11. Image signals which are detected in the imaging device (CCD)11 are fed to the image processing unit 30, so that the signals areconverted from analog signals to digital signals and then subjected topredetermined signal processing. An image capturing operation of thestereo-image capturing device and an image data recording operation forthe recording medium M are controlled in accordance with the operationsat the operation switch group 34. Although there is merely one cameralens introduced for capturing stereo image, the camera lens should bedesigned in a specific and complex shape and further performed with alot of limited devices.

The above-mentioned apparatus are performed for capturing stereo imageby means of using several calibrated lens, wherein the apparatus arelarger and more complicated. Therefore, image-calibrating is consideredto achieve the purpose of 3D image reconstruction. The 3D imagereconstruction method includes the steps of registering an orthographicimage of a scene, combining a photogrammetric image and a technicaldrawing of the scene to form a co-registered orthographic andperspective (COP) image, and reconstructing a 3D image from the COPimage. However, the photogrammetric image should be taken by severalcameras at first.

In U.S. Pat. No. 6,724,930, it discloses a three-dimensional positionand orientation sensing apparatus. Please refer to FIG. 2. Itillustrates a block diagram for showing a structure of athree-dimensional position and orientation sensing apparatus accordingto U.S. Pat. No. 6,724,930. As illustrated in FIG. 2, a plurality ofmarkers 2 (hereinafter to be abbreviated as code markers) having uniquegeometric characteristics are disposed on or near an object of whichthree-dimensional position and orientation is to be estimated. Thesecode markers 2 are photographed by an image acquisition apparatus 3, anda photographed image 5 is transferred to within a computer 4. Inprinciple, the object 1 and the image acquisition apparatus 3 have theirown coordinate systems, and the image 5 acquired by the imageacquisition apparatus 3 is defined as a camera image plane. However, theimage 5 should be further dealt by the computer. After the computer 4has received the image 5, the computer 4 extracts a candidate regionthat is estimated to be a region corresponding to the code marker 2,from within the image 5. The computer 4 analyzes in detail the candidateregion extracted, and then computes geometric characteristicscorresponding to the code of the code marker 2 from the candidateregion. When the code has been recognized, the computer registers theposition within the image and the code by recognizing this region as themarker region. Finally, the computer 4 calculates a three-dimensionalposition and orientation of the object 1 with respect to the imageacquisition apparatus 3, by utilizing the two-dimensional image positionof the code marker 2 extracted from the image registered at the step 2and the three-dimensional position of this code marker 2 with respect tothe object 1. Meanwhile there is a lot of operation introduced incomputer analyzing.

However, in practice, the prior art should perform 3D depth-capturing bymeans of introducing more than one camera, a lot of complex calibratedlens or a lot of program operation of computer. It is difficult toimplement. Hence, it needs to provide a system and method for obtainingobject depth through digital signal processing, which provides disparityvectors and camera extrinsic parameters for obtaining the depth isobtained from a disparity to depth conversion module, simplifies theentire structure and process, is capable of achieving the purpose ofautomatically obtaining object depth without change of the cameraitself, thereby facilitating user to take stereo image, and can rectifythose drawbacks of the prior art and solve the above problems.

SUMMARY OF THE INVENTION

This paragraph extracts and compiles some features of the presentinvention; other features will be disclosed in the follow-up paragraph.It is intended to cover various modifications and similar arrangementsincluded within the spirit and scope of the appended claims, and thisparagraph also is considered to refer.

Accordingly, the prior art is limited by the above problems. It is anobject of the present invention to provide an auto depth-of-fieldcapturing system of a camera for obtaining object depth through digitalsignal processing, which provides disparity vectors and camera extrinsicparameters for obtaining the depth is obtained from a disparity to depthconversion module, simplifies the entire structure and process, iscapable of achieving the purpose of automatically obtaining object depthwithout change of the camera itself, thereby facilitating user to takestereo image, and can rectify those drawbacks of the prior art and solvethe above problems.

In accordance with an aspect of the present invention, the autodepth-field capturing system for a camera includes a camera lens fortaking plural images; a camera calibration device including an epipolarestimation module for estimating plural epipolar data of the pluralimages; and a camera extrinsic parameter estimation module forestimating a location data; at least a frame buffer for storing theplural images temporarily; an image rectification module for rectifyingthe plural images corresponding to plural epipolar data and obtainingplural rectified images; a disparity estimation module connected withthe camera calibration device and the image rectification module forreceiving the location data and obtaining disparity vectors of therectified images; a disparity to depth conversion module for obtaining adepth in response to the disparity vectors and the location data; and adepth image painting module for painting a 3D image in correspondencewith the depth.

Certainly, the plural images can be taken from the camera at differentpositions.

Preferably, the plural epipolar data are plural epipolar lines andepipolar points.

Preferably, the epipolar estimation module further produces a matrix inresponse to the plural epipolar lines and epipolar points by means oftracing algorithm.

Preferably, the matrix includes a relative motion vector and a relativedirection vector between at least two of the plural images.

Preferably, the location data is the 3D position and angle of thecamera.

It is another object of the present invention to provide an autodepth-field capturing method for obtaining object depth through digitalsignal processing, which provides disparity vectors and camera extrinsicparameters for obtaining the depth through a disparity to depthconversion module and simplifies the entire structure and process. It iscapable of achieving the purpose of automatically obtaining object depthwithout change of the camera itself, thereby facilitating user to takestereo image, and can rectify those drawbacks of the prior art and solvethe above problems.

In accordance with the aspect of the present invention, the autodepth-field capturing method for a camera includes the steps of a)taking plural images; b) estimating plural epipolar data of the pluralimages for obtaining a matrix; c) estimating a location data in responseto the plural epipolar data and the matrix; d) rectifying the pluralimages corresponding to plural epipolar data for obtaining pluralrectified images; e) calculating the location data for obtainingdisparity vectors of the rectified images; f) obtaining a depth field inresponse to the disparity vectors and the location data; and g) paintinga 3D image in correspondence with the depth field.

Preferably, the step a) is executed via a camera lens.

Preferably, the plural images are taken from the camera at differentpositions.

Preferably, the step b) is executed via an epipolar estimation module.

Preferably, the plural epipolar data are plural epipolar lines andepipolar points of the plural images.

Preferably, the fundamental matrix is obtained in response to the pluralepipolar lines and epipolar points by means of tracing algorithm.

Preferably, the fundamental matrix comprises a relative motion vectorand a relative direction vector between at least two of the pluralimages.

Preferably, the step c) is executed via a camera extrinsic parameterestimation module.

Preferably, the location data is the 3D position and angle of thecamera.

Preferably, the step d) is executed via an image rectification module.

Preferably, the step e) is executed via a disparity estimation module.

Preferably, the step f) is executed via a disparity to depth conversionmodule.

Preferably, the step g) is executed via a depth image painting module.

Preferably, the auto-depth-field capturing method further includes astep of b1) providing at least a frame buffer for storing the pluralimages temporarily.

The above objects and advantages of the present invention will becomemore readily apparent to those ordinarily skilled in the art afterreviewing the following detailed description and accompanying drawings,in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram showing the electrical constructionof the stereo-image capturing device according to the prior art;

FIG. 2 illustrates a block diagram for showing a structure of athree-dimensional position and orientation sensing apparatus accordingto the prior art;

FIG. 3 illustrates an auto depth-field capturing system for a cameraaccording to the present invention;

FIG. 4 illustrates a camera for an auto depth-field capturing systemaccording to the present invention;

FIG. 5 further illustrates a display screen of the camera for thepresent invention; and

FIG. 6 illustrates an auto depth-field capturing method for a cameraaccording to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention discloses a system and method for obtaining objectdepth through digital signal processing, and the objects and advantagesof the present invention will become more readily apparent to thoseordinarily skilled in the art after reviewing the following detaileddescription. The present invention needs not be limited to the followingembodiment.

Please refer to FIG. 3. It illustrates an auto depth-field capturingsystem for a camera according to the present invention. As shown in FIG.3, the capturing system of the present invention includes a camera lens401 for taking plural images; a camera calibration device 41 includingan epipolar estimation module 411 for estimating plural epipolar data ofthe plural images; and a camera extrinsic parameter estimation module412 for estimating a location data; at least a frame buffer 413 forstoring the plural images temporarily; an image rectification module 42for rectifying the plural images corresponding to plural epipolar dataand obtaining plural rectified images; a disparity estimation module 43connected with the camera calibration device 41 and the imagerectification module 42 for receiving the location data and obtainingdisparity vectors of the rectified images; a disparity to depthconversion module 44 for obtaining a depth field in response to thedisparity vectors and the location data; and a depth image paintingmodule 45 for painting a 3D image in correspondence with the depthfield.

In practice, the plural images are taken from the camera at differentpositions respectively. After at least two images are taken via thecamera lens 401, the image will be stored into the frame buffer 413temporarily. The images stored in the images buffer 413 are introducedto the epipolar estimation module 411 of the camera calibration device41 in turn. In this embodiment, the plural epipolar data are pluralepipolar lines and epipolar points. The epipolar estimation module 411produces a matrix, such as the matrix consisting of the rotation matrixR and translation matrix t in the following equation

$\begin{bmatrix}u \\v \\1\end{bmatrix} = {{{\begin{bmatrix}f_{u} & s & c_{x} \\0 & f_{v} & c_{y} \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0\end{bmatrix}}\begin{bmatrix}R^{T} & {{- R^{t}}t} \\0_{3}^{T} & 1\end{bmatrix}}\begin{bmatrix}X \\Y \\Z \\1\end{bmatrix}}$

in response to the plural epipolar lines and epipolar points by means oftracing algorithm, where the matrix includes a relative motion vectorand a relative direction vector between at least two of the pluralimages. For example, the fundamental matrix could be used as the matrixand described as: F˜K₂ ⁻¹EK₁ ⁻¹, where K₁ and K₂ are camera parametersof two images taken in different angles. For real-time operation, pluralepipolar lines should be obtained via global estimation and then thematrix is obtained. Thus, the efficiency is increasing. The obtainedmatrix will be transmitted to the camera extrinsic parameter estimationmodule 412.

After receiving the plural epipolar lines and the matrix, the cameraextrinsic parameter estimation module 412 could produce the locationdata of the camera, wherein the location data is 3D position and angleof the camera, and is relative to the origin of the previous image. Thelocation data is supplied for a disparity estimation module 43 and adisparity to depth conversion module 44.

The image rectification module 42 rectifies the plural imagescorresponding to plural epipolar data and at least two images, andobtains plural rectified images, wherein epipolar lines of two imagesare rectified into the same one; and one center, called capturing-imagecenter, is given for the removed two images. The rectified images andthe location data of the camera extrinsic parameter estimation module412 are inputted to the disparity estimation module 43. In the presentinvention, the epipolar estimation module 411, the camera extrinsicparameter estimation module 412, and the frame buffer 413 could beintegrated in a camera calibration device 41, which is a part of an ICdevice at small size, instead of introducing a large computer.

After the disparity estimation module 43 receives the rectified imagesand the location data of the camera extrinsic parameter estimationmodule 412, the disparity vectors are produced to transmit to thedisparity to depth conversion module 44. The disparity to depthconversion module 44 will produce a depth in response to the disparityvectors and the location data, and then a depth image painting module 45is able to paint a 3D image in correspondence with the depth.

Accordingly, the present invention could be executed via a camera asshown in FIG. 4. It illustrates a camera for an auto depth-fieldcapturing system according to the present invention. As shown in theFIG. 4, the digital camera 50 includes the above described system of thepresent invention. Furthermore, there is a camera lens with an imagesensor 501 displaced on the digital camera 50. When a user partly pushesa shutter button 502 thereof, the digital camera 50 will begin toexecute the process of capturing 3D images. The result could bedisplayed on the 2D or 3D screen (not shown in FIG. 5) of the digitalcamera 50. The digital camera 50 could be moved or rotated in severaldirections 60 for capturing more images. When the user has satisfied theresult, the user could completely push the shutter button to finishtaking 3D images, and the result will be outputted or stored. FIG. 5further illustrates a display screen of the camera for the presentinvention. The display screen 70 could further include a marked region701 to define a focus region as the most important target for performingthe capturing process of the present invention.

In accordance with the aspect of the above system, the present inventionfurther provides an auto depth-field capturing method for a camera.Please refer to FIG. 6. It illustrates an auto depth-field capturingmethod for a camera according to the present invention. According to theFIG. 3 and FIG. 6, the method includes the steps of a) taking pluralimages via a camera lens 401, as shown in the procedure S601, whereinthe plural images are stored temporarily in a frame buffer 413; b)estimating plural epipolar data of the plural images via an epipolarestimation module 411 for obtaining a fundamental matrix, as shown inthe procedure S602; c) estimating a location data in response to theplural epipolar data and the fundamental matrix via a camera extrinsicparameter estimation module 412, as shown in the procedure S603; d)rectifying the plural images corresponding to plural epipolar data viaan image rectification module 42 for obtaining plural rectified images,as shown in the procedure S604; e) calculating the location data via adisparity estimation module 43 for obtaining disparity vectors of therectified images, as shown in the procedure S605; f) obtaining a depthin response to the disparity vectors and the location data via adisparity to depth conversion module 44, as shown in the procedure S606;and g) painting a 3D image in correspondence with the depth via a depthimage painting module 45, as shown in the procedure S607.

In practice, the plural images are taken from the camera at differentpositions respectively. After at least two images are taken via thecamera lens 401, the image will be stored into the frame buffer 413temporarily. The images stored in the images buffer 413 are introducedto the epipolar estimation module 411 of the camera calibration device41 in turn. In this embodiment, the plural epipolar data are pluralepipolar lines and epipolar points. The epipolar estimation module 411produces a fundamental matrix in response to the plural epipolar linesand epipolar points by means of tracing algorithm, wherein thefundamental matrix includes a relative motion vector and a relativedirection vector between at least two of the plural images. Accordingly,the method of the present invention could achieve the purpose ofautomatically obtaining object depth without change of the cameraitself, thereby facilitating user to take stereo image.

In conclusion, the present invention provides an auto depth-fieldcapturing system of a camera for obtaining object depth through digitalsignal processing. It provides disparity vectors and camera extrinsicparameters for obtaining the depth from a disparity to depth conversionmodule, simplifies the entire structure and process, and is capable ofachieving the purpose of automatically obtaining object depth withoutchange of the camera itself, thereby facilitating user to take 3D image.It can also rectify those drawbacks of the prior art and solve the aboveproblems.

While the invention has been described in terms of what is presentlyconsidered to be the most practical and preferred embodiment, it is tobe understood that the invention needs not be limited to the disclosedembodiment. On the contrary, it is intended to cover variousmodifications and similar arrangements included within the spirit andscope of the appended claims, which are to be accorded with the broadestinterpretation so as to encompass all such modifications and similarstructures.

1. An auto depth-field capturing system for a camera comprising: acamera lens for taking plural images; a camera calibration deviceincluding an epipolar estimation module for estimating plural epipolardata of said plural images, and a camera extrinsic parameter estimationmodule for estimating a location data; at least a frame buffer forstoring said plural images temporarily; an image rectification modulefor rectifying said plural images corresponding to said plural epipolardata and obtaining plural rectified images; a disparity estimationmodule connected with said camera calibration device and said imagerectification module for receiving said location data and obtainingdisparity vectors of said rectified images; a disparity to depthconversion module for obtaining a depth in response to said disparityvectors and said location data; and a depth image painting module forpainting a 3D image in correspondence with said depth.
 2. The autodepth-field capturing system according to claim 1, wherein said pluralimages are taken from said camera at different positions.
 3. The autodepth-field capturing system according to claim 1, wherein said pluralepipolar data are plural epipolar lines and epipolar points.
 4. The autodepth-field capturing system according to claim 3, wherein said epipolarestimation module further produces a matrix in response to said pluralepipolar lines and epipolar points by means of tracing algorithm.
 5. Theauto depth-field capturing system according to claim 4, wherein saidmatrix comprises a relative motion vector and a relative directionvector between at least two of said plural images.
 6. The autodepth-field capturing system according to claim 1, wherein said locationdata is 3D position and angle of said camera.
 7. An auto depth-fieldcapturing method for a camera comprising the steps of: a) taking pluralimages; b) estimating plural epipolar data of said plural images forobtaining a matrix; c) estimating a location data in response to saidplural epipolar data and said matrix; d) rectifying said plural imagescorresponding to said plural epipolar data for obtaining pluralrectified images; e) calculating said location data for obtainingdisparity vectors of said rectified images; f) obtaining a depth inresponse to said disparity vectors and said location data; and g)painting a 3D image in correspondence with said depth.
 8. The autodepth-field capturing method according to claim 7, wherein said step a)is executed via a camera lens.
 9. The auto depth-field capturing methodaccording to claim 7, wherein said plural images are taken from saidcamera at different positions.
 10. The auto depth-field capturing methodaccording to claim 7, wherein said step b) is executed via an epipolarestimation module.
 11. The auto depth-field capturing method accordingto claim 7, wherein said plural epipolar data are plural epipolar linesand epipolar points of said plural images.
 12. The auto depth-fieldcapturing method according to claim 11, wherein said fundamental matrixis obtained in response to said plural epipolar lines and epipolarpoints by means of tracing algorithm.
 13. The auto depth-field capturingmethod according to claim 7, wherein the said matrix comprises arelative motion vector and a relative direction vector between at leasttwo of said plural images.
 14. The auto depth-field capturing methodaccording to claim 7, wherein said step c) is executed via a cameraextrinsic parameter estimation module.
 15. The auto depth-fieldcapturing method according to claim 7, wherein said location data is 3Dposition and angle of said camera.
 16. The auto depth-field capturingmethod according to claim 7, wherein said step d) is executed via animage rectification module.
 17. The auto depth-field capturing methodaccording to claim 7, wherein said step e) is executed via a disparityestimation module.
 18. The auto depth-field capturing method accordingto claim 7, wherein said step f) is executed via a disparity to depthconversion module.
 19. The auto depth-field capturing method accordingto claim 7, wherein said step g) is executed via a depth image paintingmodule.
 20. The auto depth-field capturing method according to claim 7,further comprising step of b1) providing at least a frame buffer forstoring said plural images temporarily.